\newcommand{\parg}{\vspace{1ex}\noindent}

\chapter{Environment Variables}

\HPCToolkit{}'s measurement subsystem decides what and how to measure
using information it obtains from environment variables.  
This chapter describes all of the environment variables that control
\HPCToolkit's measurement subsystem. 

When using
\HPCToolkit{}'s \hpcrun{} script to measure the performance of
dynamically-linked executables, \hpcrun{} takes information passed
to it in command-line arguments and communicates it to \HPCToolkit{}'s
measurement subsystem by appropriately setting environment variables.
To measure statically-linked executables, one first adds \HPCToolkit's
measurement subsystem to a binary as it is linked by using \HPCToolkit's
\hpclink{} script. Prior to launching a statically-linked binary that
includes \HPCToolkit's measurement subsystem, a user 
must manually set environment variables.

Section~\ref{user-env} describes
environment variables of interest to users. Section~\ref{system-env}
describes environment variables designed for use by \HPCToolkit{}
developers. In some cases, \HPCToolkit's developers will ask a user
to set some of the environment variables described in Section~\ref{system-env} to generate a detailed error 
report when problems arise.

\section{Environment Variables for Users}
\label{user-env}

\paragraph{HPCTOOLKIT.}
Under normal circumstances, there is no need to use this environment variable. 
However, there are two situations, however, \hpcrun{}
\emph{must} consult the \verb+HPCTOOLKIT+ environment variable to determine the location
of \HPCToolkit{}'s top-level installation directory:

\begin{itemize}
\item On some systems, parallel job launchers (e.g., Cray's aprun) \emph{copy} the
       \hpcrun{} script to a different location. In this case, for \hpcrun{} to find libraries
       and utilities it needs at runtime, you must set the \verb+HPCTOOLKIT+ environment variable to 
       \HPCToolkit{}'s top-level installation directory.
\item
       If you launch the \hpcrun{} script via a file system link,
       you must set \verb+HPCTOOLKIT+ for the same reason.
\end{itemize}


\paragraph{HPCRUN\_EVENT\_LIST.}

This environment variable is used provide a set of (event, period)
pairs that will be used to configure \HPCToolkit's measurement subsystem to perform 
asynchronous sampling.  The HPCRUN\_EVENT\_LIST environment variable
must be set otherwise HPCToolkit's measurement subsystem will terminate
execution. If an application should run with sampling disabled,
HPCRUN\_EVENT\_LIST should be set to NONE. Otherwise, HPCToolkit's
measurement subsystem expects an event list of the form shown below.
$$event1[@period1];...;eventN[@periodN]$$ As denoted by the
square brackets, periods are optional. The default period is 1
million.

\parg
Flags to add an event with \hpcrun: \verb|-e/--event| $event1[@period1]$ 

\parg
Multiple events may be specified using multiple instances of \verb|-e/--event| options.

\paragraph{HPCRUN\_TRACE.}

If this environment variable is set, HPCToolkit's measurement
subsystem will collect a trace of sample events as part of a measurement
database in addition to a profile. HPCToolkit's hpctraceviewer
utility can be used to view the trace after the measurement database
are processed with either HPCToolkit's hpcprof or hpcprofmpi
utilities.

\parg
Flags to enable tracing with \hpcrun: \verb|-t/--trace|

\paragraph{HPCRUN\_OUT\_PATH}

If this environment variable is set, HPCToolkit's measurement subsystem
will use the value specified as the name of the directory where
output data will be recorded. The default directory for a command
$command$ running under control of a job launcher with as job ID
$jobid$ is hpctoolkit-$command$-measurements[-$jobid$]. (If no
job ID is available, the portion of the directory name in square
brackets will be omitted. Warning: Without a $jobid$ or an output
option, multiple profiles of the same $command$ will be placed
in the same output directory.

\parg
Flags to set output path with \hpcrun: \verb|-o/--output| $directoryName$

\paragraph{HPCRUN\_PROCESS\_FRACTION}
\sloppy
If this environment variable is set, \HPCToolkit's measurement
subsystem will measure only a fraction of an execution’s processes.
The value of HPCRUN\_PROCESS\_FRACTION may be written as a a floating
point number or as a fraction.  So, '0.10' and '1/10' are equivalent.
If HPCRUN\_PROCESS\_FRACTION is set to a value with an unrecognized
format, \HPCToolkit's measurement subsystem will use the default
probability of 0.1. For each process, \HPCToolkit's measurement
subsystem will generate a pseudo-random value in the range [0.0, 1.0).
If the generated random number is less than the value of
HPCRUN\_PROCESS\_FRACTION, then \HPCToolkit{} will collect performance
measurements for that process.

\parg
Flags to set process fraction with \hpcrun: \verb|-f/-fp/--process-fraction| $frac$

\paragraph{HPCRUN\_MEMLEAK\_PROB}

If this environment variable is set, \HPCToolkit's measurement
subsystem will measure only a fraction of an execution’s memory
allocations, e.g., calls to \verb|malloc|, \verb|calloc|, \verb|realloc|,
\verb|posix_memalign|, \verb|memalign|, and valloc. All allocations
monitored will have their corresponding calls to free monitored as
well. The value of HPCRUN\_MEMLEAK\_PROB may be written as a a
floating point number or as a fraction.  So, '0.10' and '1/10' are
equivalent. If HPCRUN\_MEMLEAK\_PROB is set to a value with an
unrecognized format, \HPCToolkit's measurement subsystem will use the
default probability of 0.1. For each memory allocation, \HPCToolkit's
measurement subsystem will generate a pseudo-random value in the range
[0.0, 1.0). If the generated random number is less than the value
of HPCRUN\_MEMLEAK\_PROB, then \HPCToolkit{} will monitor that
allocation.

\parg
Flags to set process fraction with \hpcrun: \verb|-mp/--memleak-prob| $prob$

\paragraph{HPCRUN\_DELAY\_SAMPLING}

\sloppy
If this environment variable is set, HPCToolkit's measurement subsystem
will initialize itself but not begin measurement using sampling
until the program turns on sampling by calling
\verb|hpctoolkit_sampling_start()|. To measure only a part of a
program, one can bracket that with \verb|hpctoolkit_sampling_start()|
and \verb|hpctoolkit_sampling_stop()|. Sampling may be turned on
and off multiple times during an execution, if desired.

\parg
Flags to delay sampling with \hpcrun: \verb|-ds/--delay-sampling|

\paragraph{HPCRUN\_RETAIN\_RECURSION}

Unless this environment variable is set, by default HPCToolkit's
measurement subsystem will summarize call chains from recursive calls
at a depth of two. Typically, application developers have no need
to see performance attribution at all recursion depths when an
application calls recursive procedures such as quicksort. Setting
this environment variable may dramatically increase the size of
calling context trees for applications that employ bushy subtrees
of recursive calls.

\parg
Flags to retain recursion with \hpcrun: \verb|-r/--retain-recursion|

\paragraph{HPCRUN\_MEMSIZE}

If this environment variable is set, HPCToolkit's measurement subsystem
will allocate memory for measurement data in segments using the
value specified for HPCRUN\_MEMSIZE (rounded up to the nearest
enclosing multiple of system page size) as the segment size. The
default segment size is 4M.

\parg
Flags to set memsize with \hpcrun: \verb|-ms/--memsize| $bytes$

\paragraph{HPCRUN\_LOW\_MEMSIZE}

If this environment variable is set, HPCToolkit's measurement subsystem
will allocate another segment of measurement data when the amount
of free space available in the current segment is less than the
value specified by HPCRUN\_LOW\_MEMSIZE. The default for low memory
size is 80K.

\parg
Flags to set low memsize with \hpcrun: \verb|-lm/--low-memsize| $bytes$


\section{Environment Variables for Developers}
\label{system-env}

\paragraph{HPCRUN\_WAIT}

If this environment variable is set, HPCToolkit's measurement subsystem
will spin wait for a user to attach a debugger. After attaching a
debugger, a user can set breakpoints or watchpoints in the user
program or HPCToolkit's measurement subsystem before continuing
execution. To continue after attaching a debugger, use the debugger
to set the program variable DEBUGGER\_WAIT=0 and then continue.
Note: Setting HPCRUN\_WAIT can only be cleared by a debugger
if \HPCToolkit{} has been built with debugging symbols.
Building \HPCToolkit{} with debugging symbols requires
configuring \HPCToolkit{} with --enable-develop.

\paragraph{HPCRUN\_DEBUG\_FLAGS}

\HPCToolkit{} supports a multitude of debugging flags that enable a
developer to log information about HPCToolkit's measurement subsystem
as it records sample events. If HPCRUN\_DEBUG\_FLAGS is set, this
environment variable is expected to contain a list of tokens separated
by a space, comma, or semicolon. If a token is the name of a debugging
flag, the flag will be enabled, it will cause HPCToolkit's measurement
subsystem to log messages guarded with that flag as an application
executes. The complete list of dynamic debugging flags can be found
in HPCToolkit's source code in the file
src/tool/hpcrun/messages/messages.flag-defns. A special flag value
“ALL” will enable all flags. 

\parg
Note: not all debugging flags are
meaningful on all architectures. 

\parg 
Caution: turning on debugging flags
will typically result in voluminous log messages, which will may
dramatically slow measurement and the execution under study.

\parg
Flags to set debug flags with \hpcrun: \verb|-dd/--dynamic-debug| $flag$

\paragraph{HPCRUN\_ABORT\_TIMEOUT}

If an execution hangs when profiled with HPCToolkit's measurement
subsystem, the environment variable HPCRUN\_ABORT\_TIMEOUT can be
used to specify the number of seconds that an application should
be allowed to execute. After executing for the number of seconds
specified in HPCRUN\_ABORT\_TIMEOUT, HPCToolkit's measurement
subsystem will forcibly terminate the execution and record a core
dump (assuming that core dumps are enabled) to aid in debugging.

\parg
Caution: for a large-scale parallel execution, this might cause a
core dump for each process, depending upon the settings for your
system. Be careful!

\paragraph{HPCRUN\_FNBOUNDS\_CMD}

For dynamically-linked executables, this environment variable must
be set to the full path of a copy of HPCToolkit's hpcfnbounds
utility. This utility is available at
\verb|/path/to/hpctoolkit/libexec/hpctoolkit/hpcfnbounds|.


